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A long-standing communications problem is the efficient coding of 
a block of binary data into a pair of in-phase and quadrature components. 
This modulation technique may be regarded as the placing of a discrete 
number of signal points in two dimensions. Quadrature amplitude modu- 
lation (QAM) and combined amplitude and phase modulation (AM-PM) 
are two familiar examples of this signaling format. Subject to a peak or 
average power constraint, the selection of the signal coordinates is done so 
as to minimize the probability of error. In the design of high-speed data 
communication systems this problem becomes one of great practical sig- 
nificance since the dense packing of signal points reduces the margin 
against Gaussian noise. Phase jitter, which tends to perturb the angular 
location of the transmitted signal point, further degrades the error rate. 
Previous investigations have considered the signal evaluation and design 
problem in the presence of Gaussian noise alone and within the framework 
of a particular structure, such as conventional amplitude and phase modu- 
lation. We present techniques to evaluate and optimize the choice of a signal 
constellation in the presence of both Gaussian noise and carrier phase 
jitter. The performance of a number of currently used or proposed signal 
constellations are compared. 

The evaluation and the optimization are based upon a perturbation 
analysis of the probability density of the received signal given the trans- 
mitted signal. Laplace's asymptotic formida is used for the evaluation. 
Discretizing the signal space reduces the optimal signal design problem 
under a peak power constraint to a tractable mathematical programming 
problem. 

Our results indicate that in Gaussian noise alone an improvement in 
signal-to-noise ratio of as much as 2 dB may be realized by using quadra- 
ture amplitude modulation instead of conventional amplitude and phase 
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modulation. New modulation formats are proposed which perform very 
well in Gaussian noise and additionally are quite insensitive to moderate 
amounts of phase jitter. 

I. INTRODUCTION 

A very attractive modulation format for coherent high-speed data 
transmission is the family of suppressed-carrier, two-dimensional signal 
constellations of which quadrature amplitude modulation (QAM) and 
combined amplitude and phase modulation (AM-PM) are two ex- 
amples. In this paper we will consider using this more general signal 
format, which is equivalent to the arbitrary placement of a discrete 
number of signal points in the plane, subject only to a peak or average 
power constraint. The object will be to mitigate the major statistical 
transmission impairments encountered on the voice-grade telephone 
channel, such as carrier frequency offset, carrier phase jitter, and 
additive noise. Our attention is focused on constellations of 16 points, 
since this seems to be the largest constellation which is practical 
for the typical telephone channel. However, the techniques we develop 
are applicable to constellations of arbitrary size. 

The placing of signal points in the plane is a long-standing problem 
that has received considerable attention in the past. Previous investi- 
gations 1-3 have considered the signal evaluation and design problem in 
the presence of Gaussian noise alone and within the framework of 
a particular structure such as combined amplitude and phase modula- 
tion. When Gaussian noise is the only transmission impairment, it is 
well known that at high signal-to-noise ratios (>25 dB) the signal 
points should be placed as far apart from each other as possible (the 
circle-packing problem). In the application to high-speed digital com- 
munication systems, the two-dimensional signal design problem be- 
comes one of great practical significance because the dense packing of 
a large number of signal points markedly reduces the margin against 
random noise and phase jitter. The signal design problem in the pres- 
ence of both phase jitter and Gaussian noise has not been solved before 
and is the subject of our discussion. 

Coherent receiver structures have recently been proposed which em- 
ploy an adaptive equalizer 4 to compensate for any linear distortion, 
low-frequency phase jitter, or small amounts of frequency offset intro- 
duced by the channel. A phase-locked loop 5 may be used to suppress 
high-frequency phase jitter and frequency offset. Of course, the output 
of a phase-locked loop will still deviate somewhat from the optimum 
demodulating phase angle. One purpose of this study is to assess the 
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effect of such phase errors on the system error rate and to indicate how 
this knowledge can be incorporated into the system design. 

By system design we have in mind the selection of both a particular 
two-dimensional signaling format and the decision device placed at the 
demodulator output. In order to pursue these objectives, we discuss: 

(i) the relative immunity of various signal constellations as a func- 
tion of the degree of noncoherency (i.e., the size of the phase 
error) ; 
(it) an efficient iterative procedure for determining optimum signal 

formats under a peak power constraint; 
(in) system performance for the following hierarchy of decision de- 
vices: easily implementable, optimum in Gaussian noise, and 
a maximum-likelihood detector which uses the statistics of the 
phase error; 
(iv) the accuracy required in an}' phase-locked loop to attain a satis- 
factory error rate; and 
(v) the resulting error rate when no attempt is made to track the 
jitter.* 

Our approach is to assume that intersymbol interference has been 
effectively eliminated by the equalizer while the phase-locked loop, if 
there is one, has only partially removed the phase jitter. Thus the 
equalizer output will be the sum of the partially coherent 1 transmitted 
signal and additive Gaussian noise. We adopt a phenomenological 
model which assumes that the (slowly varying) phase error has 
a Tikhonov density. 5 The Tikhonov density is associated with a con- 
ventional first-order phase-locked loop whose input is the sum of a 
sinusoid (whose phase is being tracked by the loop) and Gaussian 
noise. Under our assumed operating conditions of high signal-to-noise 
ratio, this density will closely approximate the actual phase density. 
When no attempt at tracking is made, 1 the jitter is modeled as being 
uniformly distributed in a reasonable peak-to-peak range. For each 
of these jitter densities, the probability density of the demodulator 
output, conditioned on the transmitted symbol, is used to estimate the 
error rate. This estimate is of the minimum distance type, where the 
"distance," which reflects the presence of phase jitter, is measured in 

* In the sequel, we will use the term jitter as a catch-all when referring to phase 
jitter and/or frequency offset. 

* A partially coherent signal is one whose carrier phase is jittered by a random com- 
ponent which is not uniformly distributed in the range ( — tt, *-). 

* Since the passband equalizer' will determine the optimum static demodulating 
phase, the absence of a phase-locked loop does not imply the use of an arbitrary de- 
modulating phase. 
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a non-Euclidean manner. The error rate is given for various signal 
constellations and detector structures under peak and average power 
constraints. By discretizing the received signal space, an iterative 
procedure is developed to determine (locally) optimum signal formats 
under a peak power constraint. This technique, which assumes a maxi- 
mum-likelihood detector, makes use of an efficient search procedure 
developed by Kernighan and Lin. 6 

The system model and the problem formulation are presented in 
Section II. An asymptotic estimate and an upper bound on the error 
rate are developed in Section III. A comparison of the relative im- 
munity of some popular signal constellations and detectors to phase 
jitter is described in Section IV. Section V discusses a technique to 
obtain locally optimum signal structures under a peak power constraint. * 

II. SYSTEM MODEL AND PROBLEM FORMULATION 

2.1 Preliminaries 

We consider the two-dimensional synchronous data communication 
system shown in Fig. 1. Binary data are first grouped into blocks of 
M bits, and each block of M bits is then mapped into one of 2 M two- 
tuples (a, b). The sequences {a k \ and [b k \ amplitude modulate, re- 
spectively, an in-phase and quadrature carrier to generate the trans- 
mitted signal 

m(t) = £ a k p(t - kT) cos u c t + £ b k p(t - kT) sin a> c t, (1) 

where 1/T is the symbol rate,* p(-) represents the transmitter pulse 
shaping, and to c is the carrier frequency. It will be assumed that the 
two-tuples are equiprobable. The received signal at the output of the 
bandpass filter is given by 

r(t) = (£ a k x(t - kT) - £ b k y(t - kT)) cos ((«, + A)* + d(t)) 

k k 

- (L a k y(t -kT) + Z b k (t - kT)) 

k k 

X sin ((« fl + A)* + 0(0) + n(t), (2) 

where x(t) and y(t) are the system (baseband) in-phase and quadrature 
impulse responses, 1 A is the carrier frequency offset, 6(t) is the random 

'The present authors have recently treated the two-dimensional signal design 
problem under an average power constraint. 7 

f Note that the data rate is M/T bits/second. 

* These pulses represent the cascade of the transmitter shaping filter, the channel, 
and the receiving filter. 
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Fig. 1 — An in-phaae and quadrature data transmission system. 

phase jitter, and n(t) is additive Gaussian noise. The received signal, 
which is sampled at the symbol rate, is adaptively equalized and 
then coherently demodulated with the aid of a phase-locked loop. 5 
The demodulated output, denoted by the sequence of two-tuples 
z* = ( (zk, Zk) ) , is then processed by the (simple) detector to give the 
output sequence \(d k , b k )\. The system error rate is just the probabil- 
ity that (d k , b k ) differs from the transmitted two-tuple (a k , b k ). 

2.2 Basic Model 

For the purposes of this study, it will be convenient to assume that 
the equalizer has completely eliminated the intersymbol interference 
present in x(t) and y(t), but that the phase-locked loop has only par- 
tially compensated for the carrier phase jitter. The in-phase and 
quadrature demodulator outputs, at the kth sampling instant, are 
then given by 



z{kT) = a k cos 4>k - b k sin <j> k + n c {kT), 
Z{kT) = a k sin (f> k + b h cos <f> k + n,(kT), 



(3) 



932 THE BELL SYSTEM TECHNICAL JOURNAL, JULY-AUGUST 1973 



• • 

• • 


b | 

• >\ A 

/ / \ 

/ 1 / 

/ \/ 

/ x i 

'/ ■ • 

// i 
// . 


• • 

• • 


• • 

• • 



Fig. 2 — Effect of Gaussian noise and phase jitter on transmitted symbol. 

where 4> k is the (slowly varying) phase error in the tracking loop, and 
n c (kT) and n,(kT) are, respectively, the in-phase and quadrature 
Gaussian noise components. * Dropping the time index, eq. (3) can be 
rewritten to give the basic model 

z = Rs + n, (4) 

where the vectors are given by f 

•-G). -(:)■ -©• 

and the matrix ft is the rotational (by an angle <f>) transformation 

B /«*>* -si"*). ( 5 ) 

\ sin 4> cos <p / 

* Recall that n c (kT) and n,{kT) are independent Gaussian random variables with 
equal variance, JVo- It should be noted that 2iVo is the noise power contained in the 
bandwidth of the received signal. 

+ We denote the values that a and 6 can assume by a<'> and 6 W) , respectively, and 
the values of the transmitted symbols by s ( '> = (o<»>,b U) ). 



TWO-DIMENSIONAL SIGNAL CONSTELLATION 933 

I. 



-3 



-3 



AVERAGE POWER = 3.00 
PEAK POWER = 4.24 



Fig. 3 — Quadrature amplitude modulation. 



As we show in Fig. 2, the effect of the phase jitter is to rotate the trans- 
mitted symbol, s, by an angle tf> ; thus the demodulator output, z, is dis- 
persed in an angular manner due to phase jitter and in a circularly sym- 
metric way due to the Gaussian noise. The receiver will make an error 
when these perturbations move the demodulator output across the 
decision boundary associated with the transmitted symbol. For a par- 
ticular transmitted data sequence, the demodulated sequence will be 
scattered about the transmitted points in a manner which reflects the 
combined effects of phase jitter and Gaussian noise. For the transmitted 
signal constellation of Fig. 3, which is known as QAM, a typical scat- 
tered demodulated sequence is shown in Fig. 4. As one might expect, 
for those signal points further away from the origin, the angular dis- 
placement becomes more apparent. An estimate of this effect is given 
by the mean-square error between the transmitted and demodulated 
symbols. For small values of jitter, this error is obtained from (4) by 
noting that 



- sO"> 



-( 



COS 0—1 

sin </> 



— sin 
cos <t> 



t\y +»-*(?) +»; 



averaging the norm-squared of both sides gives* 
E\\z - s<»|| 2 = iV + 4\\aw\\*, 



(6) 



'The notation ||z|| denotes the Euclidean norm of z; additionally the notation 
(z, s) will be used to denote the inner product of z and s. 
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Fig. 4 — Received signal points in a QAM system. 

where a\ is the variance of <f> and E denotes the statistical average. 
Thus, because of phase jitter, signal points located further from the 
origin are subjected to a larger mean-square error. 

2.3 Probability Density of the Demodulator Output 

In order to evaluate the system error rate, the probability density 
function (pdf) of the phase error must be specified. The pdf of the phase 
error in a phase-locked loop that is tracking the angle of the two- 
dimensional data signal given by (2) is not yet known, but as explained 
below it can be approximated by the following (Tikhonov) density 5 : 

1 p a cos <t> 

«*>-frm> l * |g " (7) 
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where 7 (-) is the modified Bessel function of the first kind and a is 
a positive number. As is shown in Fig. 5, a = implies a completely 
incoherent system, while a = x corresponds to a completely coherent 
system. For large values of a (a > 100), we have the useful relation 

a\ ~ - (in radians). (8) 

Since the above density arises from a first-order phase-locked loop 
whose input is the sum of Gaussian noise and a sinusoid (whose phase 
is being tracked by the loop), it is felt that for high signal-to-noise 
ratio, for negligible intersymbol interference, and for slowly varying 
phase jitter the actual phase-error density will closely resemble the 
Tikhonov density. This simple model will be quite useful in studying 
the effect of phase jitter on the system error rate. 

The pdf of the demodulator output, conditioned on the transmission 
of s ij) , is given by 

p(z[s<») A p .(z) =y%,-(z|0)pfo)d0, (9) 

where it is noted that the output density conditioned on both 8 U) and 
<f> is given by 



P ' (z| * )= 2^F eXp 



mil--*"!! 1 ! < 10a ) 



exp j- 2^r o [||z - B w||« + 2<z, s<>>) 



2vNo 

-2<z, i2s">)]j. (10b) 
Substituting the Tikhonov density into (9) gives 

P ' (Z) " 2^-o 6XP {" 2To [I|Z " 8l ^ + 2<Z ' S<>)) + 
(/o(a))- 1 i j* exp i- [«z, B W) + aN ) cos <f> 

+ <z, b«>a) sin 0]d0, (11) 

(where if s U) = ||s 0) ||(cos a, sin a) then s ( ' } * = ||s W) ||(sina, -cosa)), 
and we recognize the latter integral as 

h (^; V((z,s^) +a Ar )2 +(ZjS oH )2 |\ . 
Assume a = fc/N (A; a constant). Employing the well-known result 
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Fig. 5— Tikhonov phase jitter density p (<*>) = exp (a cos $)/2ir/ (a). 

that for large values of argument h(x) « e x / 1 Vz| , and assuming a is 
sufficiently large and N Q is sufficiently small, we get 



p/z) 



27rJVoV 



aNt 



exp 



( 2AT ( 



V«z,S<>>)+aJVo) 2 + «z,S<^)) 2 
X {||z, s u) \\ 2 + 2<z, S<») + 2aN 

- 2V«z, b«> + cxNo) 2 + (z, B»u)i} J. (12) 

In the error rate computations we shall eventually make, we will have 
k so large compared to the practical range of (z, s 0) > that the coeffi- 
cient multiplying the exponential of p,(z) can be taken to be (2ttN )- 1 . 
Thus, for our purposes, 



P /z)»^exp[-^(z,s<<>)], 
where 
d 2 (z, s) ± ||z - s|| 2 + 2<z, s) + 2aiV 



(13) 



- 2| V||z|| 2 ||s|| 2 + 2aN (z, s> + (aN ) 2 \. (14) 

The form of eq. (13) is very reminiscent of the density in Gaussian 
noise alone, and to further suggest such a similarity we refer to d(z, s) 
as the "distance" between z and s. It is important to emphasize that 
this function is the key to assessing the combined effect of Gaussian 
noise and phase jitter on the system performance; through its use we 
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SNR = 25 dB, AND RMS PHASE JITTER = 9 DEGREES 
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Fig. 6 — Constant distance contours : values of z for which 

2wN oPi (z) = exp{- ^^(z, *>)} = 10 " 2 - 10 ~ 6 > 10_, °- 
d 1 given by eq. (14). 



are able to give a very suggestive geometric interpretation of the jitter 
phenomenon as well as accurate estimates of the error rate. In Fig. 6 
we show some contours of constant "distance" (i.e., points which are 
equiprobable) about a given point. Note the angular orientation and 
similarity of these "banana" shaped contours to those obtained ex- 
perimentally (Fig. 4). 

Our procedure will be to use (13) and (14) to estimate the error rate 
for various constellations and detector structures; however, before we 
do this, we first wish to discuss some properties of the function d(z, s) 
which will be useful in estimating the system performance, and then to 
consider the conditional density p,{z) in the absence of a phase-locked 
loop. 

2. 4 Some Properties of the Jitter Distance d{z, s) 

(i) A requisite property that d(z, s) should possess is that for vanish- 
ingly small jitter the distance between points becomes Euclidean. This 
is easily verified by expanding (14) in terms of 1/a and observing that 



lim d 2 (z, s) = ||z — s| 

l/a-0 



(15) 
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(u) Since we do not expect the distance between an arbitrary point 
and the origin to be affected by phase jitter, it is reassuring to note that 

d»(z,0) = ||z|| 2 , (16) 

and that a similar property holds for points on the same ray, i.e., 

d*(z, fez) = (1 - fc) 2 ||z|| 2 , k > 0. (17) 

(Hi) A "bonus" property is that even though (14) was derived for 
large a (small jitter), the expression 

limd 2 (z,s) = | ||z|| - ||s|| | 2 (18) 

a-*0 

is quite reasonable in that, for completely incoherent systems, all 
points on the same circle are indistinguishable. 

(iv) An interesting consequence of (14) is that points are now closer 
together than their Euclidean distance. To demonstrate this, we use 
the Schwartz inequality to show that the non-Euclidean part of d 2 (z, s) 
is always negative, i.e., 

(z, s) + aN g | V||z|| 2 ||s|| 2 + 2aN (z, s) + (aN ) 2 \ (19) 

since squaring gives 

<Z, S) 2 + 2aN (z, s) + (ccNo) 2 ^ ||z|| 2 ||s|| 2 + 2aN (z, S> 4- (aiVn) 2 

which upon cancelling becomes the Schwartz inequality 

(z, s) 2 ^ ||z|| 2 ||s|| 2 . 

(v) If both the signal and noise power are scaled by the same con- 
stant for a given jitter level, a, it is clear that the conditional density is 
unchanged. Thus the signal-to-noise ratio and the mean-square jitter 
a are the natural parameters for characterizing the system. 

(vi) A natural question to ask is whether or not d(z, s) is a convex 
metric* in the plane or, of more practical interest, if some sufficiently 
accurate approximation to d(z, s) is a convex metric in some circle 
centered about the origin. As we shall see later, if such an approximate 
representation can be obtained, some tedious error rate computations 
may be done quite simply. While the requirements of symmetry and posi- 
tivity can easily be shown to hold in the entire plane, J. E. Mazo has re- 
cently informed us that his results in Ref. 8 imply that d(z, s) is not a 

' A convex metric is a metric which possesses the midpoint property, i.e., for any 
two points x and y there is always a third point z such that d(x, z) = d(z, y) 
= (l/2)d(x, y). 
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convex metric in any circle about the origin for any value of 2N$. The 
question remains open as to whether or not d(z, s) admits an accurate 
convex metric approximation in a neighborhood of the origin. Appendix 
B reports the results of an investigation of this question. 

By considering the above properties, it is apparent that a signal 
constellation will be relatively immune to small amounts of phase jitter 
either if the error rate (or minimum distance) is determined by a point 
at the origin and any other signal point, or if signal points on the same 
circle are widely separated (the more circles the greater the Gaussian 
noise penalty). 

2.5 Probability Density in the Absence of a Phase-Locked Loop 

In a subsequent section we will compare the performance of various 
signal constellations, and the following question naturally arises: Can 
we, by judicious signal design, eliminate the need for a phase-locked 
loop ? * Preliminary to answering this question we must obtain the den- 
sity of the demodulated signal in the absence of a phase-locked loop. 
For simplicity we model the jitter as arising from the single tone 

<f>(t) = A cos M + #), (20) 

where 2 A is the peak-to-peak jitter, «,- is the jitter frequency, and f is 
a uniformly distributed random phase. For this model, the jitter den- 
sity is given by 

1 Ul ^ A 



p(<t>) = 



l -VRl) 



(21) 
\<f>\ > A 



To determine the density of the demodulated samples we use (9) and 
(10) to write 

^ = ^ ex p {- m tu* - s0) ii 2 + 2 <*> ■«>:} 

XJ_ A exp -j^r [(z, iW) cos <f> + (z, s ( >">) sin 0]l p(0)d« 
which for small (<12 degrees) peak-to-peak jitter becomes 

"' (Z) - 2SV„ ex P [ " 2F„ « Z - S " , » I ] ***>, (22) 



* The equalizer 4 is capable of determining the optimum demodulating static phase, 
so that the demodulation is noncoherent only to the degree that the untracked jitter 
degrades the error rate. 
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where M+( • ) is the moment-generating function of <j>, and 

fe = ± (z, ■«*). (23) 

For the jitter density given by (21) it is easy to show that 

MMd = JoUlfcl); (24) 

thus we have the familiar form 



where 



^-sm^l-m 1 ***")]' (25) 



D*(z, s) = ||z - s|| 2 - 2# &i /o ^ | (z, b«*) | J 



(26) 



It is useful to summarize our work up to this juncture. Using the 
simple model (4) and the phase-error densities (7) and (21), we have 
derived the conditional density of the demodulated two-tuple with 
and without a phase-locked loop. Equations (13) and (14) and (25) 
and (26) are the desired expressions. In the next section we will use 
these densities to estimate the system error rate. 

III. ESTIMATING THE ERROR RATE 

In this section we give two estimates of the error rate : an asymptotic 
(high SNR) evaluation and an upper bound. Consider the arbitrary 
signal constellation shown in Fig. 7, where the decision regions are de- 
noted by Rj. The detector, which is specified by the decision regions, 
will declare that s U) has been transmitted if and only if the demodu- 
lated vector z falls inside Rj. Because of various practical considera- 
tions, principally ease of implementation, the mathematically optimum 
detector will not always be the one which is built. 

3. 1 Asymptotic (High SNR) Error Rate 
The probability of error is given by 

P e = £ P,Pe„ (27) 

where the p/s are the (taken to be equal) a priori probabilities and P '«,- 
is the conditional error rate. The conditional error rate is just the prob- 
ability that z falls outside R, when s (i) is transmitted, i.e., 

P ej = Pr [z $ Rj\sM transmitted]. (28) 
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Fig. 7 — Typical decision region about signal point s,-, with minimum distances to 
boundary shown for Gaussian noise alone (no phase jitter) and for large jitter (angu- 
lar displacements highly likely). 

This quantity may be written, using the conditional density p,(z), as 



P 



- -/. 



£*/ 



Pi(z)dz. 



(29) 



Let z* denote a point of global minimum for the function d 2 or D 2 on 
the decision region boundary and let M, be the number of times this 
minimum is achieved on the boundary. Let (u, v) denote an orthogonal 
coordinate system erected at z* with the positive u axis pointed along 
the boundary line in a clockwise direction and v pointed outside Rj. 
Then, as is shown in Appendix A, for a high signal-to-noise ratio 
(A^o — * 0) with aNo = k, the conditional error rate is given by 



Pei = 




•exp 



-d- 

2N n 



(30a) 



- z j 



For the case of Gaussian noise alone (no phase jitter) d 2 becomes the 
ordinary Euclidean distance and the partials in (30a) are easily evalu- 
ated. In this case, with s (,) the signal(s) closest to s (y) , we have 



z, = 



S (0 -J_ S U) 
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and 



dd* 

dv 

d 2 d 2 
du 2 



z=z' 



= l|s (i) - s">| 



= 2. 



Hence, for the Gaussian case, 

My [No - j|sw -s»>||» ,, ftM 

** = 2 ||,(o - sO)|| V V exp 8A^ ' (3 ° b) 

a useful formula in its own right. 

In terms of the distance functions d(-) and D(-), the asymptotic 
error rate is determined by the point on the decision boundary "closest" 
to the transmitted signal. Of course, in the presence of phase jitter, 
this point will generally differ from the closest point according to 
a Euclidean measurement. In Fig. 7, the indicated point on the vertical 
boundary segment is the closest point to By in the Euclidean sense. 
As phase jitter increases, those points with a radial coordinate nearly 
equal to that of By becomes closer to By. So for large phase jitter, 
the point indicated on the boundary segment above s ; is the "closest" 
point. Thus the exponential decay in error rate is quite similar to the 
asymptotic Gaussian result since it is of the form 

exp [-dSun(y)/2iVo], (3D 

where d m i D (j) is the minimum distance (measured in a non-Euclidean 
manner) to the jth decision boundary. The minimum distance can 
sometimes be obtained analytically, but most often must be obtained 
by a computer search of the boundary. 

The minimum distance to the jth decision boundary is particularly 
easy to determine if the function d( -, •) is a metric which possesses the 
midpoint property, i.e., for each distinct pair of points s (,) and s U) 
there exists a third point z* such that 

|d(s<«, ■«) = d(s«>, z # ) = d(z„ ■«). 

If this is the case, let z, denote any point on the decision boundary be- 
tween s (i) and s 0) and the triangle inequality gives 

d(B«>, z.) + d(s«>, Zi) ^ d(s<« s<»). (32a) 

Since for maximum-likelihood detection 

d(S«>, Zi ) = d(S<« Zi), 
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d(s<», z*) ^ min $d(s (0 , s">) 



fo«i 



(32b) 



where z* is the point on the boundary closest to s 0) . Note that (32b) 
would provide an upper bound on (31). Since the midpoint z* clearly 
lies on the boundary, z* = z*, and (32b) is satisfied with equality, 
the minimum distance is given by 



d(s<«>,z*) = min fd(s<*>, s<*). 



(32c) 



Since the Euclidean metric is convex, eq. (30b) could be directly 
obtained from (30a) by using (32c). 

3.2 An Upper Bound on the Error Rate (for Small Jitter) 

In systems which use a tracking loop, an upper bound on the system 
error rate may be obtained, for small jitter, by considering Fig. 8. This 
figure shows the transmitted point s 0) , the decision boundary R jt and 
several nested regions C, (denned by contours of constant probability), 
where 

Cj = {z:d 2 (z,s<») ^ cj}. 




Fig. 8 — Nested equidistance contours [distance denned by eq. (13)3 about signal 
Si in arbitrary decision region. Contour c,-*, at distance <£,-*, defines ck* as the shortest 
distance to the boundary. 
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If we let C* denote the first contour which touches the boundary, then 
it is clear that 

P ej = Pr [z $ # y |s<>> transmitted] ^ Pr [z £ C*|s ( »] 

J z&C* } 

= [ exp [ - n 4r d 2 {z, s<») 1 rfz. (33) 

For small jitter, the exponent may be expanded in the first power of 
1/a, to give 

d*(x, s) = ||x - s|| 2 - ~ (x, s 1 ) 2 , (34) 

for which the contours of constancy are ellipses. Transforming the 
ellipses into circles and changing to polar coordinates enable us to inte- 
grate (33) to get 

P ej ^ \ exp [-dXWo]. (35) 

V 1 aNo 



Again diSu is the minimum distance to the jth decision boundary which 
for convex polygonal decision boundaries can be determined analyti- 
cally. For high SNR, the above bound is useful up to 1.5 degrees rms 
jitter. Because of the similarity of (30a) and (35), we will use only the 
former asymptotic results in the sequel. 

IV. A COMPARISON OF VARIOUS SIGNAL CONSTELLATIONS 

In the preceding section we have presented a means of evaluating the 
asymptotic (high SNR) error rate for a given signal constellation and 
detector structure. In terms of the minimum distance, measured via 
the appropriate noise/phase-jitter distance function to the jth decision 
boundary, we have 

(36) 



e *~*f*> 2||s<« - s<>>|| V tt exp L 2JV ^J 

The minimum distance will be obtained by a computer search of the 
decision boundary. It should be emphasized that comparisons based on 
the asymptotic error rate are not exact but rather indicate order-of- 
magnitude effects. 
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Our comparisons will be made by varying the following quantities : 

(f) signal constellations 

(it) signal-to-noise ratios 

(in) rms jitter 

(iv) decision boundaries 

(y) phase-error density. 

Clearly the pie may be sliced several ways, so let us first say a few 
words about each of the above variables. 

(i) Signal constellations: For the purposes of signal evaluation we 
will consider the existing 16-point constellations QAM, 8-8, and (4, 90°) 
shown in Figs. 9a through 9c. The circular constellation (4, 90°) has 
signal points equally spaced (i.e., 90 degrees apart) on four circles. This 
large angular spacing of points on the same circle suggests that this 
constellation will be insensitive to small amounts of phase jitter. The 
ratio of outer radius to inner radius (r 2 /ri) for the 8-8 constellation is 
1.59, found by Lucky 1 to minimize the error rate in Gaussian noise. 
8-8 is an optimized form of AM-PM in which signal points on the outer 
circle do not lie on the same radial lines as those on the inner circle. It 
offers an order-of-magnitude improvement (over AM-PM) in error rate 
in the presence of Gaussian noise. We will also consider the new circu- 
lar modulation formats 1-5-10, 1-6-9, 5-11, shown in Fig. 10. The 
optimum ratio (r 2 /ri) is very close to 2 for these constellations as we 
have determined by equating the three smallest nearest-neighbor 
distances. 

(ii) We consider peak and average SNR's, P pk and P aV g respec- 
tively, chosen so that P e (no jitter) is < 10~ 6 . These quantities are 



f 



i-3 1-1 



-t — t 



.-4- 
i 




i_i=a .__!__ ___i 



Fig. 9 — Existing signal constellations: (a) quadrature amplitude modulation 
(QAM), ppe a k/p a v = 1.8; (b) modified AM-PM (8-8), Ppeak/Pav = 1.43; (c) circular 
constellation, (4, 90°) ppeak/pav = 1.85. 
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Fig. 10 — New signal constellations: (a) 1-5-10, ppesk/pa 

Ppeak/Pay = 1.525 J (c) 5-11, Ppeak/W = 1.31. 



= 1.42; (b) 1-6-9, 



defined by 



P pk = max||s<»|| 2 /2Ar , 



Pn 



1 18 

= — T 

i6,ti 



.(»||2 



/2N . 



(Hi) Our attention is focused on the practical range of residual 
(from a PLL) jitter of <3 degrees rms. 

(iv) We will consider both the boundaries which are optimum in 
Gaussian noise (straight lines) as well as more practical boundaries for 
the circular formats (polar wedges). 

(i») The Tikhonov density will be taken as representative of those 
systems which use a tracking loop, while the peak-to-peak density will 
be used to model the raw (untracked) phase jitter. 

The error rate curves are grouped as follows: Figure 11 shows the 
error rate vs rms residual phase jitter (Tikhonov density) under aver- 
age and peak power constraints for the six constellations described 
above. For each constraint, a Gaussian noise power (and thus an SNR) 
is assumed which places the curves in a useful operating range, and the 
receiver is presumed to use the Gaussian optimum decision region 
boundaries. These boundaries are piecewise-linear contours constructed 
from segments of perpendicular bisectors of lines joining signal points as 
shown in Fig. 12a for the 1-5-10 constellation. This is equivalent to 
deciding in favor of the signal point closest in Euclidean distance to the 
demodulated point. Figure 12b shows a more "practical" set of deci- 
sion boundaries for the 1-5-10 constellation. Figure 11 indicates the 
immunity of the (4, 90°) constellation to small amounts of phase jitter 
at the expense of an error rate more than an order of magnitude greater 
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SNR = 23dB 



2.5 3 0.5 1 

RMS JITTER IN DEGREES 



1.5 



2.5 



Fig. 11 — Error rate vs jitter with a phase-locked loop: (a) average power con- 
straint; (b) peak power constraint. 

than those of the 1-5-10, 1-6-9, and QAM constellations. In practical 
operation with a tracking loop, any of these alternative constellations 
will almost always outperform the (4, 90°) constellation. 

Figure 13 shows the error rates vs SNR, again under average and 
peak power constraints, in the presence of Gaussian noise alone (no 
phase jitter) and with 1.5 degrees rms residual phase jitter in addition 





(b) 



Fig. 12 — Decision region boundaries for part of 1-5-10 constellation: (a) Gaussian- 
optimum decision region boundaries ; (b) practical decision region boundaries. 
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Fig. 13 — Error rate vs SNR for channels with and without phase jitter; Gaussian- 
optimum receiver is assumed to have a phase-locked loop : (a) no jitter, average power 
constraint; (b) no jitter, peak power constraint; (c) rms jitter = 1.5 degrees, average 
power constraint; (d) rms jitter = 1.5 degrees, peak power constraint. 
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Fig. 14 — Probability of error for receiver with "practical" decision region bound- 
aries for the 1-5-10 constellation and with phase-locked loop: (a) average power con- 
straint; (b) peak power constraint. 

to the Gaussian noise. Although some of the curves shift their relative 
positions (at least under the average power constraint) when phase 
jitter is added, the good performances of 1-5-10 and 1-6-9 are main- 
tained. QAM performs respectably and the (4, 90°) constellation comes 
in last. 

Figure 14 is similar to a reduced Fig. 11 except that the "practical" 
set of decision region boundaries is presumed for the 1-5-10 constella- 
tion. As can easily be seen, the Gaussian optimum boundaries for QAM 
are also practical boundaries, and the jitter-immune (4, 90°) constel- 
lation is shown to best advantage by presuming Gaussian optimum 
boundaries. Under the average power constraint, QAM is superior to 
1-5-10 below about 1.5 degrees rms jitter. Under the peak power con- 
straint, 1-5-10 is uniformly superior to QAM. The (4, 90°) constella- 
tion does not show an advantage until the rms jitter reaches 2.5 to 3 
degrees. 

Figure 15 is a set of error rate vs SNR plots with practical decision 
region boundaries for 1-5-10. Curves are shown for Gaussian noise 
alone (no jitter) and 1.5 degrees rms jitter. Only the average power 
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Fig. 15— Probability of error vs SNR for Gaussian-optimum receiver with "practi- 
cal" decision region boundaries for the 1-5-10 constellation and with phase-locked 
loop: (a) no jitter, average power constraint; (b) rms jitter = 1.5 degrees, average 
power constraint. 

constraint is presumed. As before, QAM shows an advantage in the 
absence of phase jitter and still does well in the presence of moderately 
severe residual phase jitter. 

Figure 16 presents some data for receivers which do not use tracking 
loops. In this case the peak-to-peak density of eq. (22) describes the 
raw jitter. Curves are plotted vs peak jitter under average and peak 
power constraints. Some interesting features are the resistance of 1-5- 
10 up to a threshold of about 8 degrees peak-to-peak jitter and the 
rapid deterioration of the performance of QAM. 

Figure 17 shows the performance vs SNR for the receivers without 
tracking loops when the peak-to-peak jitter is 12 degrees. This is the 
only instance for which the (4, 90°) constellation looks relatively good, 
but here, too, the 1-6-9 constellation performs slightly better. QAM, of 
course, does rather poorly. 

The advantage of using a phase-locked tracking loop with QAM and 
1-5-10 can be seen from the above data and a simple calculation. If 
the raw phase jitter is modeled by 

Q{t) = A cos lujt + *1 
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Fig. 16— Probability of error vs. peak-to-peak jitter for Gaussian-optimum receiver 
without a phase-locked loop : (a) average power constraint ; (b) peak power constraint. 

where \p is uniformly distributed from - tt to tt and 2 A is the peak-to- 
peak jitter, then a rule of thumb 9 suggests that the residual rms jitter 
out of a tracking loop is of the order of 0.1 X 2A. For A = 6 degrees, 
this rms value is 1.2 degrees. A comparison of the curves of Figs. 11 and 
16 shows that substantially lower error rates are achieved by the re- 
ceiver with a tracking loop. The performance of constellation (4, 90°) 
is relatively unaffected by the introduction of a tracking loop. 

A further conclusion that can be drawn from the numerical data is 
that the QAM constellation, which is simple to generate and to de- 
modulate, performs quite well in Gaussian noise alone or (with the aid 
of tracking loop) Gaussian noise plus phase jitter. More circular con- 
stellations, such as 1-5-10 and 1-6-9, appear to offer a moderate fur- 
ther advantage at the expense of greater complexity. 

V. OPTIMUM SIGNAL CONSTELLATIONS UNDER A PEAK POWER CONSTRAINT 

In this section we discretize the received signal space to obtain a 
tractable optimization problem. The discretizing is such that the M 
signal points are selected from a circle containing L points while the 
received points lie in a circle of N (N > L) points (note: M < L). We 
make the following two comments concerning this approach to solving 
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24.5 



Fig. 17 — Probability of error vs SNR for Gaussian-optimum receiver without a 
phase-locked loop: (a) peak-to-peak jitter = 12 degrees, average power constraint; 
(b) peak-to-peak jitter = 12 degrees, peak power constraint. 



the problem : 

(i) the level of discretization must be fine enough to provide a good 
approximation to the continuous problem, and 

(tt) the outer radius must be chosen so that for all practical purposes 
the probability that a received point lies outside the outer circle is 
negligible. The peak power constraint simply means that no signal 
points can be selected outside the L circle. 

5. 1 Discrete Maximum-Likelihood Formulation 

If we denote the received point, z, by "i" and the transmitted signal, 
s°\ by "j," then the perturbation of the transmitted signal due to 
Gaussian noise and phase jitter may be summarized by a transition 
matrix whose elements are defined by 

p(i\ j) = Pr [receiving "i" | transmitting "j"] 

i = l, 2, ..-N (37) 

j= 1,2, ■■■M. 

The transition probabilities may be computed by integrating the condi- 
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tional densities p/(z) over an appropriate region. It is convenient to 
work with the maximum-likelihood receiver (i.e., the optimum decision 
boundaries are used) which receives "i" and declares that "A" was 
transmitted, where 

p(i|A) > p(i\j), j * A j, A = 1, 2, • • -M. (38) 

Note that we have (temporarily) fixed the M points in the signal con- 
stellation. It is easy to see that the probability of being correct is given 
by 

N 

Pr [correct] = £ Pr [correct | receive "»"] Pr [receive "«"], (39) 
,•=1 

but by Bayes rule 

Pr [correct | receive 'V] 

= Pr [send "A" | receive "t"] 

_ Pr [receive "t"|send "A"] Pr [send "»"] 



Pr [receive ' V] 



(40) 



Substituting (40) in (39) and recalling that the transmitted signals 
are equiprobable gives 

Pr [correct] = ^ f) Pr [receive "»"|send "A"] 

-i£lK.m (41) 

and the optimum constellation is the M signals (or columns) that 
maximize 

£pW<). (42) 

Note that since p(i|A) is the maximum entry in the ith row of the 
transition matrix, the problem is one of selecting M out of L columns 
such that the sum of the row maxima is maximized. The error rate may 
be determined from (41). 

5.2 Optimum Constellations 

A heuristic program for solving the combinational optimization 
problem posed by (42) has been developed by Kernighan and Lin. 6 
Their process is based upon iterative improvement of either known 
initial constellations or random initial starts. For each start, a "locally 
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optimum" solution is found in the sense that no change of position of 
a single signal can improve the criterion. The heuristic process is very 
fast and, for the resolution we require, 20 random starts can be pursued 
to completion in 25 seconds. For particular values of rms phase jitter 
and noise power we find, among the best of 20 local optima, reaffirma- 
tion of known solutions and in some instances new competitive con- 
stellations. For example, for a peak signal energy of SNR = 22 dB, 
Figs. 18, 19, and 20 give the best among the 20 local optima for an 
rms of 0, 1.5, and 3 degrees, respectively. As expected, the 0-degree 
solution has a 5-11 character and the 1.5-degree solution has a 1-5-10 
character. On the other hand, the 3 -degree solution is somewhat of 
a surprise; it has a 1-6-9 character. The (4, 90°) constellation, which is 



Fig. 18 — Optimum signal constellation: Peak SNR = 27 dB, no jitter (courtesy of 
B. W. Kernignan and S. Lin). 
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Fig. 19 — Optimum signal constellation: Peak SNR = 27 dB, 1.5 degrees rms jitter 
(courtesy of B. W. Kernighan and S. Lin). 



best at 3 degrees among heretofore considered designs, has an error 
rate only a few percent worse than that of the 1-6-9 constellation. 

A byproduct of the development of the above procedure is the 
demonstration of the fact that numerical quadrature routines offer 
a competitive alternative to asymptotic techniques and bounding 
methods for the estimation of system error rates. 



VI. CONCLUSIONS 



Comparisons have been made of several well-known two-dimen- 
sional signal formats in the presence of Gaussian noise and phase 
jitter, at high signal-to-noise ratios and under both peak and average 
power constraints. It has been demonstrated that, under an average 
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Fig. 20— Optimum signal constellation : Peak SNR = 27 dB, 3 degrees rms jitter 
(courtesy of B. W. Kernighan and S. Lin). 

power constraint for systems which have a high-quality phase-locked 
loop (rms residual jitter <1 degree), QAM had the lowest error rate 
of all candidate constellations. If the residual jitter is <1.5 degrees 
rms, the 1-5-10 constellation becomes extremely attractive since it is 
immune to phase jitter in this range and provides the same asymptotic 
(no-jitter) error rate as QAM. For small amounts of jitter, 1-5-10 and 
QAM have a 2-dB SNR advantage over the (4, 90°) constellation which 
is "immune" to phase jitter. Both these constellations offer a 0.5-to-l- 
dB advantage in SNR over conventional AM/PM signaling techniques. 
Thus, under an average power constraint, both QAM and 1-5-10 merit 
consideration. 

For a peak power constraint, in addition to making comparisons 
similar to the above, we have been able to attain the optimum signal 
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constellations for various levels of jitter. Our comparisons indicate 
that, for jitter <1 degree, QAM suffers a 1.5-dB SNR penalty with 
respect to the 5-11 constellation, while 1-5-10 suffers a 0.1-dB penalty. 
At 1.5 degrees rms jitter, 1-5-10 again is superior to both QAM and 
5-11 (by 4 and 1 dB respectively). 

Based upon the peak and average power constraints, the new modu- 
lation format 1-5-10 appears to make extremely efficient use of avail- 
able signal power and for a slight increase in modulation/demodula- 
tion complexity offers considerable immunity to moderate (<1.5 
degrees rms) residual phase jitter. QAM systems which employ high- 
quality phase-locked loops will also be operating very efficiently, pro- 
vided that the residual phase error is <0.8 degree rms. 
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APPENDIX A 

Method of Laplace 

Let g(x) and h(x) be continuous real functions on [a, 6] where h(x) 
is also twice continuously differentiable. Then, if h(x) attains a single 
maxima at c (a < c < b), we have that 

(read ~ as asymptotic to) . The proof is not difficult and the key steps 
can be found in Papoulis 10 or in Jones. 11 This analysis technique for 
estimating an integral for large values of the parameter k is called the 
method of Laplace. 

An immediate application of this method used in the body of this 
paper is 

/o(a) A J- [' e"^ 8 dd ~ -?L («->«>). 

For estimating system error rates, a certain two-dimensional version 
of Laplace's method is needed. Particularly, we shall investigate the 
following two-dimensional integral: 



/ ± 

.£0 



/ / exp 1 k h( ^ \ dz t z = (w, »)] 
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for small k. The function h(z) we shall be concerned with is assumed 
to have the following properties : 

(i) h(z) is twice continuously differentiate such that h v (0) < 0, 
h uv (0) ^ 0, and is the unique point of maximum for h(z) in 
v ^ 0.* Thus we also assume h uu of (0) < 0. 
LetG r = \v ^ 0, u 2 + u 2 ^ r] 
(ii) For some r > 0, 

Rewriting I as 

e u/*>M0) J f exp|^[A(z) -/i(0)]i, 

( O g0) 

it is easy to conclude that for each e > 

/ ^ g(l/*)A( 



i( °iifo exp {l Lh{z) ~ A(o)] 



We shall proceed to integrate with the exponent in the integrand re- 
placed by its local representation 



e d/*)A(o) /*' f' eX p Ij- [h t (0)v + h uu (0)u 2 /2 + h uv (0)uv] > dvdu. 

The absence of h v (0) and h vv (0) follow directly from (i). Integrating 
(dv) we get 

■'-' [ fe,(0) + ft„,(0)u J 

X e (l/fc(A uu (0)(uJ/2)^ w 

We appeal to the first paragraph to integrate each term involved in 
this subtraction. Take e small enough to avoid the singularity at 
u = —h v (0)/h uv (0). The first term is asymptotic to 

* 3 ' ! \SHh )+Mo)e -^} 



*In this appendix, subscripts are used to denote partial derivatives, e.g., 
dh 

(U, B)-(0. 0) 



*■« - ev 
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while the second is 

A,(0) V^ uu (0) 
Since for e small enough h v (0)e — [A^(0)//i uu (0)]« 2 < 0, we conclude 

MO) \A U u(0) 

In error rate computations one is often integrating over the exterior 
of a convex polygon. In the body of this paper we encounter the case 
where h(z) has a finite number of global maxima on the boundary, at 
most one on each side, and none at the vertices. Let z* n be the nth local 
maximum. Map the exterior half-space containing z n into the upper- 
half plane via a rotation, composed with a translation taking z* n — > 0. 
Then the method of the last paragraph can be applied. The process is 
repeated for each maxima and the results sum to the required asymp- 
totic estimate of the exterior integral. The fact that the exterior half- 
spaces containing distinct points of maxima may overlap is of no conse- 
quence. Furthermore, in our applications, h(z) is symmetric with re- 
spect to each z* and the process need only be completed once and the 
answer multiplied by the multiplicity of the maxima. 

APPENDIX B 

The Nature of d{x, y) 

B.l Introduction 

Let V be a vector space endowed with a scalar product (x, y) and 
a norm derived therefrom. It is not known whether d y (x, y) : V X V — > 
\R\ defined by 



d y (x,y) = (||x|| 2 + ||y|| 2 + 2 7 -' -2|V||x|| 2 ||y|| 2 + 2 7 - 1 (x,y) + 7- 2 l) 1/2 

is a metric for any values of y (0 < y < °o). Nor is it known whether 
d y (x, y) is a metric in any sphere centered about the origin. Similarly, 
the status of the midpoint property is also unknown to us. However, 
from Ref. 8, we know that these two properties cannot hold simultane- 
ously for any y in any sphere about the origin. 

Concerning the metric question, the difficulty (as usual) is the 
triangle inequality. By definition d y (x, y) is symmetric in its arguments. 
Positivity follows easily from the Schwartz inequality, 

d y (x,y) ^ {llx -yl| 2 + 2(x > y) + 2 7 - 1 

" 2|V||x|| 2 ||y|| 2 + 2 7 -'||x||||y||-r-7- 2 |} = I IWI - ||y|| I ^ 0. 
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Notice d y (x, y) > unless x = y. If d y (x, y) is a metric on V X V for 
a particular value of y then it is a metric for all y (0 < y < *> ) ; this is 
easily obtained by employing the mapping z — > | Vy|z. 

B.2 One-Dimensional Case 

In the special case 7 = .ft we can show di(x, y) is a metric. In one 
dimension we have (the "1" subscript will now be suppressed) 

f I x - y | xy +1^0 

( ' T ' V) = " I | V(.r + 2/) 2 + 2 2 | .ry + 1 ^ 0* 
To show the triangle inequality, first notice that, if three points a, b, 
and c are on the same side of zero, the distances are all Euclidean. So 
for the remaining cases to be investigated we assume one point has 
a different sign than the other two. Notice d(x, y) = d(-x, -y) so we 
lose no generality by assuming c ^ b ^ ^ a. Two subcases remain : 
(i) only d(c, a) is non-Euclidean; (ii) d(c, a) and d(b, a) are non- 
Euclidean. For (i) the distances involved are (c - b), (b — a), and 
| [(a + c) 2 + 2 2 ] 1/2 1 . Now | [(a + c) 2 + 2 2 ] 1 ' 2 1 ^c- a since, by squar- 
ing, this is equivalent to 1 + ac ^ 0. To show 

(c _ 6 ) ^ (6-o)+ |[(a + c) 2 + 2 2 ;r| 
and 

(6 - o) * (e - 6) + | [(« + c) 2 + 2 2 ] 1 ' 2 ! , 

it is enough to show 

[(c + a) - 26] 2 ^ [(a + c) 2 + 2 2 ] 
since squaring both sides when the right-hand side is positive can only 
weaken the inequality. The last inequality can be simplified to 
2 _ oc ^ i _|_ ah for which the left-hand side is negative and the right- 
hand side is not. On the other hand, (ii) is immediate since c — b, 
V(a + c) 2 + 4, and V(6 + a) 2 -f- 4 can be identified as sides of a tri- 
angle with apex (-a, 2) and base points c and 6. Notice d does not 
have the midpoint property since d(10, -10) = 2, yet the only points 
y satisfying d(10, y) = 1 are 9 and 11, but both d(9, -10) 
d(ll, -10) exceed 2. 

This last observation shows that the open spheres in this metric 
space are not all connected. In two dimensions, the boundaries of cer- 
tain open spheres are disconnected; specifically, it can be shown that 
for certain values of q > 

\y\d(x,y) = q\ 
is a disconnected set if ||y|| 2 7 > 1- 
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B.3 Approximating d y (x, y) on the Unit Circle 

As mentioned in the text, an important open question is whether 
d y (x, y) can be accurately approximated by a convex metric. There 
are, of course, many ways in which one can approximate d y (x, y). In 
this section we dispose of two approximations which suggest themselves 
immediately. 

Let us view d y (x, y) on the unit circle. Notice, as 7 — > 0, 



|V||x|| 2 ||y|| 2 + 2 7 - 1 <x,y) + 7- 2 l 

^-(Hr(,y) + r f tfl|y|l ;- (x ' y) > 

Hence, for small phase jitter, 

d 2 y (x, y) « ||x - y|| 2 - 7l|x|| 2 ||y|| 2 sin 2 0, 
where 8 is the angle between x and y, or what is the same, 

d y (x,y) «|| x -y||(l-47p^[ 2 ) 1/2 , 

where A(x, y) is the area of the triangle Oxy. Let h xy denote the altitude 
of Oxy perpendicular to xy; then 

d y (x,y) « ||x-y||(l -yh%yi\ 
Since h £ 1, 

\\x -y||(l -yh 2 xv yi 2 ^ 

with equality if and only if x = y so long as 7 < 1 ; also, the left-hand 
side is symmetric. Notice 

A(x,y) = ||x-y||(l -*yw 

has the midpoint property. Indeed, if z lies on the line segment joining 
x and y, 

d(x, z) + d(z, y) = d(x, y) 

since h xt = h xy = h yt . 

So we are strongly motivated now to see if A(x, y) is a metric on the 
unit disc. Note that if F is any finite subset of the unit disc, then for 
7 sufficiently small, d(x, y) is a metric on F. This follows from the fact 
that for noncolinear triples, the triangle inequality holds properly for 
the metric ||x — y||. Perhaps surprisingly, the triangle inequality for 
A 7 (x, y) does not hold for all triplets in the unit disc. This was dis- 
covered by linearizing A(x, y) to get 8(x, y) = \\x — y||(l — y/2h 2 xv ) 
which is likewise a valid approximation for d(x, y) for small 7. 
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For <5(x, y), the midpoint property and all requirements for a metric 
hold except for the triangle inequality. The triangle inequality question 
is easier for us to investigate for 5(x, y) than for A(x, y) as it de- 
velops into a geometrical extremal problem which we can handle. In 
solving the extremal problem, a class of vector triplets in the unit disc 
emerge for which no value of 7 > exists such that the triangle in- 
equality holds uniformly for the class. Although these triplets arise in 
the analysis of 8(x, y), they serve just as ruinously for A(x, y). 

In order to discuss this geometric extremal problem, we require some 
notation. Suppose we have a triangle (nontrivial — positive area) in the 
unit disc with side lengths a, b, and c and opposing vertices A, B, and 
C, respectively. Let h a , h b , and h c denote the distance from the origin 
to the line containing the side of length a, b, and c, respectively. 

Write the "triangle inequality" expression for 8(x, y) : 

l|x-y||(i-£>4)i ||x-z|| (1-^/4) 

+ l|i-y||(i-g*i)- 

Changing to the notation just introduced, we have directly that the 
triangle inequality holds for ^ 7 ^ y where y = inf Y(a, b, c). 



T ^ b ' c)= \ahl + bhl-ch\ 



and the infimum is over all triangles for which the bracketed expression 
is positive. This geometric extermal problem is disposed of by substitut- 
ing a triangle of the form depicted in Fig. 21. It follows easily 
lim e _o r = 0. Thus y = and the only d(x, y) metric is the obvious 
one. A straightforward substitution of the three vectors depicted 
above into the "triangle inequality" for A(x, y) yields again that for 
no fixed 7 > can the triangle inequality hold for this family of 
triangles. 

Strikingly, for the class of triplets in Fig. 21, it is easily demonstrated, 
via substitution, that the triangle inequality for d y (x, y) holds for an 
open interval including 7 = 0! 

B. 4 A Metric on the Boundary of the Disc 

We end on a positive note: S(x, y) is a metric on the boundary of the 
unit disc for a nontrivial interval [0, y~\.* It is enough to show inf 



* We anticipate applications in phase-modulation systems. 
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Fig. 21 — Infimizing family of vectors. 



r(o, b, c) > where the infimum is over those triangles circumscribed 
by the unit circle for which r is positive. Now hi = 1 — a 2 /± and 
similarly for hb and h c so 



Dividing gives 






(a, &, c) = (l - i L 2 + b* + c> -ab + ac + bc- - _f a6c _ - j) '" 
So y > if and only if 



sup 



abc 



a -\- b — c 



< 00. 



At this point, we must digress and recall some plane geometry from 
Ref. 12. First abc/ 4b is the area of the triangle and 2~ 1 (a + b + c) is 
called a semiperimeter. Given any triangle ABC, extend the two lines 
emanating from the apex A. Construct bisectors to the two exterior 
angles complementary to the angles B and C respectively. The bisec- 
tors meet in a point equidistant from the three lines containing a, b, 
and c. The circle tangent to these three lines is called an excircle. See 
Fig. 22. The center of the excircle (where the bisectors meet) is called 
the excenter and of course the radius is called the exradius. Each tri- 
angle has three excircles. Finally from Ref. 12 we need: 
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Fig. 22 — Excircle tangent to c. 




UPPER BOUND 
ON EXRADII 



Fig. 23 — Upper exradius bound for excircles tangent to c. Since ^ c ^ 2, the 
exradii are uniformly bounded. 
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Theorem: An exradius of a triangle is equal to the ratio of the area to the 
difference between the semi-perimeter and the side to which the excircle is 
tangent internally. 

So if the set of exradii of triangles circumscribed by the unit circle is 
uniformly bounded away from infinity, then y > and {5 Y | < 7 < Y are 
metrics. 

To complete the proof, consider any triangle with vertices on the 
unit disc boundary. With an isometric transformation of the circle into 
itself, we can situate c horizontally with apex C above c in the left 
half-plane. The more obtuse the exterior angles at A and B, the larger 
the excircle tangent to c. So take a horizontal line a' through B and a 
line b' going through A and (0, 1) and replace a and 6 with a' and b' . 
Clearly, bisectors constructed using a' and b' will intersect at a point 
further away from c than the excenter of any excircle tangent to c. By 
similar triangles, the primed bisectors intersect at a distance 
2(1 ± Vl — c 2 /4) from c where the sign is plus if c lies in the lower 
half-plane and negative otherwise (Fig. 23). 

REFERENCES 

1. Lucky, R. W., and Hancock, J. C, "On the Optimum Performance of N-ary 

Systems Having Two Degrees of Freedom," IEEE Trans. Commun. Syst. 
(March 1962). 

2. Salz, J., Sheehan, J. R., and Paris, D. J., "Data Transmission by Combined AM 

and PM," B.S.T.J., 50, No. 7 (September 1971), pp. 2399-2419. 

3. Thomas, C. M., "Amplitude Phase-Keying with Af-ary Alphabets: A Technique 

for Bandwidth Reduction," Int. Telemetering Conf. Proc, VIII (October 
1972). 

4. Gitlin, R. D., Ho, E. Y., and Mazo, J. E., "Passband Equalization of Differen- 

tially Phase-Modulated Data Signals," B.S.T.J., 52, No. 2 (February 1973), 
pp. 129-238. 

5. Viterbi, A. J., Principles of Coherent Communication, New York: McGraw-Hill, 

1966, pp. 86-96 and 118-119. 

6. Kernighan, B. W., and Lin, S., "Heuristic Solution of a Signal Design Optimiza- 

tion Problem," Proc. Seventh Annual Princeton Conf. on Information Sciences 
and Systems, March 1973. 

7. Foschini, G. J., Gitlin, R. D.. and Weinstein, S. B., "Optimization of Two- 

Dimensional Signal Constellations in the Presence of Gaussian Noise," to be 
published in IEEE Transactions on Communications. 

8. Mazo, J. E., "A Note on Metrics and Metric Convexity," unpublished work. 

9. Falconer, D. D., unpublished work. 

10. Papoulis, A., The Fourier Integral and Its Application, New York: McGraw-Hill, 

1962. 

11. Jones, D. S. "Asymptotic Behavior of Integrals," SIAM Review, 14, No. 2 

(April 1972) pp. 286-317. 

12. Davis, D. R., Modern College Geometry, Reading, Mass. : Addison- Wesley, 1957. 



